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Abstract: This article discusses the importance of measuring how students perceive quality of service in online 
higher education. The article also reviews the existing literature on measuring users’ perceptions about quality in 
e-services. Even when there are a lot of articles on this matter, none of them focuses on e-learning services, so 
this paper tries to fill that gap. The article proposes using the Critical Incident Technique to perform a qualitative 
analysis, which contributes to identify the main dimensions and categories that contribute to students’ perception 
of service quality. A case study, regarding a completely online university, is presented and the proposed model 
is used to obtain some preliminary research results. Among these, key quality dimensions from a student point of 
view are identified. Some of these dimensions are: learning process, administrative processes, teaching 
materials and resources, etc. After discussing the research results, a list of recommendations for university 
managers is formulated. We believe that both the proposed methodology and the case-study recommendations 
can be of potential interest for managers of several universities offering online higher-education worldwide. 
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1. Introduction and motivation 

Universities worldwide must face, among other challenges, an increasingly differentiated demand for 
education, the need to carry out more commercial activities in order to obtain new sources of funding, 
and new competitors that make use of Information Technologies (ITs) to offer their educational 
services in a global market. All together, these factors are forcing universities to rethink their 
traditional roles, to develop new organizational structures and to reposition themselves through 
strategic direction setting (Moratis and van Baalen 2002). These trends and the widespread 
recognition that the university’s invisible product, knowledge, is the most important factor in economic 
and social growth are the reasons for the increasing competitiveness inside the higher education 
market all over the world. To be successful in this environment, universities should focus on 
customers’ perceptions of service quality -understanding ‘service’ in the broad sense, including both 
academic and non academic services- since those perceptions are a key influence on students’ 
decisions when they are choosing or recommending a particular institution. 

While there is little disagreement on the importance of service quality issues in higher education, the 
challenge is to identify and implement the most appropriate measurement tools in order to gain a 
better understanding of the quality issues that impact on students’ service experiences (O’Neill and 
Palmer 2004). In other words, knowing what customers expect is the first and possibly most critical 
step in delivering quality (ZeithamI and Bitner 2003). 

Following the general pattern set by service industries, the issue of service quality within the higher 
education sector has received increasing attention in recent years. The most dominant theme is the 
development of valid, reliable and replicable measures of perceived service quality (PSQ). In the 
early stages, most models designed to evaluate PSQ focused exclusively on teaching and learning. 
In the last decade, though, several studies have approached the evaluation of university services from 
a broad perspective, considering not only the core service but the peripheral or auxiliary 
administrative and backup services as well (Abdullah 2005). 

This study continues this line of research by applying a holistic conception of service quality in online 
higher education. 
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2. Related work and added value of our approach 

The recent literature describes measurement tools and techniques for assessing PSQ within the 
higher education sector. For the most part they are extensions or adaptations of SERVQUAL models 
(Buttle 1996), where service quality is the result of a comparison between expectations and 
perceptions of performance, e.g.: SERVPERF (Oldfield and Baron 2000) or IPA (Ford et al. 1999, 
Joseph et al. 2005). While those models were initially designed to be applicable across a broad 
spectrum of service settings, many studies have stressed that the industry-specific characteristics of 
many services mean that these models should be adapted or supplemented to fit the characteristics 
of the particular service under analysis (Cox and Dale 2001, Chen 2002). Given these 
considerations, the relatively large number of articles on the subject of evaluating PSQ in higher 
education is in contrast to the almost total absence of such work with regard to higher-education in 
online environments. 

The digital nature the interactions produced in an online environment is a source of some problems 
for applying the classical PSQ evaluation models: 

■ Most of the items used in these scales are linked to the direct interpersonal interaction that 
characterized ‘traditional’ services. Therefore, even those who advocate the use of these scales 
in virtual environments acknowledge that, in the absence of these traditional interactions, the 
scales need to be adapted to the specific e-service context (van Riel et al 2001). 

■ The absence of physical reference points or indicators of quality of service, such as premises, 
facilities, and service staff. In the traditional university these tangible elements make up what is 
known as the ‘servicescape’ which is a decisive factor in PSQ evaluations (Q’Neill and Palmer 
2004). In online learning environments, the student does not have at his/her disposal the 
conventional physical elements that act as indicators of the quality of service. In their place, the 
student can only use other variables, such as the aesthetics and ease of use of the online 
interface, referred to as ‘e-scape’ (van Riel et al. 2004). 

■ Students are not just users of university services, but are the universities’ primary customers. 
Consumers are often part of the production and levier processes of services (Gronroos 1990), but 
in online higher-education the role of the student is even greater, since it is essential that he/she 
should be the centre of the teaching/learning process. Moreover, as the user of a digital interface, 
the online student will need a certain degree of skill and experience in working with ITs (Juan et 
al. 2008). Consequently, the students themselves contribute directly to the quality of service 
delivered and to his/her own satisfaction or dissatisfaction. 

Although there are many models for assessing PSQ in online environments are available, they have 
been designed exclusively to assess service quality of web sites and, specifically, of online shopping 
sites. The aforementioned were not designed to evaluate the quality of pure and complex services, 
such as the educational ones, which do not involve just a single transaction, but multiple interactions 
that take place over a prolonged time span. Specifically, in the case of online higher education, 
important questions, both empirical and theoretical, have just begun to be addressed. Most of the 
published studies focus on specific services -e. g. an online university library (Q’Neill 2003)- or on 
particular sections -as, for instance, teaching resources (De Lange et al. 2003)-, but to date no 
holistic evaluation of PSQ, that captures the online student’s overall service experience in online 
learning environments has been carried out. 

3. Research scenario: The Open University of Catalonia (UOC) 

The Qpen University of Catalonia or UQC (http://www.uoc.edu/portal/english) is a fully online 
university with headquarters in Barcelona, Spain. It was founded in 1995 by the Catalan Government 
with the mission of “providing people with lifelong learning and education through intensive use of 
information and communication technologies”. According to official data, the UQC offers educational 
services over the Internet to more than 50,000 students, distributed in several undergraduate and 
graduate programs. Figure 1 shows the distribution of students in Bachelors Degrees (23,671), 
Diplomas (16,593), Qpen Programmes (8,712), etc. 

UQC students belong to different parts of the world, but they are mainly located in Spain and South 
America. About 60% of UQC undergraduate students are adult students (over 30 years old) that 
typically combine their professional activity and/or family responsibilities with their academic duties. 
Educational services are delivered by a team composed of more than 2,200 instructors -including 
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UOC faculty and UOC online collaborators, most of these professors from other Spanish universities- 
and 550 management staff. The UOC uses an asynchronous and student-centred educational model 
and has already received several international prizes, such as the 2001 ICDE Prize for the best virtual 
and distance university in the world or the 2004 OEA Prize for educational quality. Currently, up to 22 
accredited degrees and official masters are offered via the UOC Virtual Campus, a learning 
management system entirely developed and maintained at the UOC (Figure 2). Some of the most 
popular degrees (in number of registered students) offered at the UOC are as follows: Computer 
Engineering, Business Administration and Management, Psychology, Telecommunications, 
Information and Communication Sciences, Law, and Humanities. 



Open programmes' 

Summer-rWinter School Courses 4,600 


University ©’thenaeum 3,472 
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Postgraduate 1,434 


Total: 54,245 


Figure 1: Distribution of UOC students by type of studies 
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Figure 2: A screenshot of the UOC Virtual Campus with some of its functionalities 


4. Research methodology 

The identification of the qualitative functional sections of service quality was carried out using a 
qualitative method referred to as the Critical Incident Technique (CIT). CIT was introduced in the 
social sciences more than fifty years ago by Flanagan (1954) and has been used in a variety of 
contexts in recent years to explore service research issues. The critical incident technique aims to 
contribute to improving our understanding of the activity or phenomenon by using an original 
approach: the reporting of the events that make up a specific experience by the person or persons 
involved. The method uses a survey or a similar procedure to obtain a catalogue of critical incidents. 


www.ejel org 


153 


ISSN 1479-4403 





Electronic Journal of e-Learning Volume 8 Issue 2 2010 (151 - 160) 


This catalogue is then compiled and analyzed to determine the key dimensions or sections that affect 
quality of service as measured by the consumers. 

The data (critical incidents) collection process comprises two stages. In the first stage, consumers 
are interviewed and specific information about the service is obtained. In the second stage, data is 
classified in categories which are intended to represent different sections of service quality. In the 
first stage of our study, an e-mail was sent to a random sample of UOC students. The theory 
recommends interviewing a minimum between 10 and 20 consumers. That way, if one of them 
provides false or mistaken information, his/her point of view can be compared and contrasted with 
data from other individuals. The selected individuals are usually asked to report between 5 and 10 
positive and 5 and 10 negative examples of their user experience regarding the analyzed service. 

We sent students an e-mail with some standard examples of critical incidents (Figure 3), so that they 
had a better idea of the kind of feedback we were expecting from them. To avoid biased responses 
as much as possible, we decided that those standard examples should be related to a health service 
instead of an academic service. We sent this e-mail to a considerably larger number of students, 
asking them to record approximately five positive and five negative critical incidents related to the 
different services they received during a complete course. We did this because the response rate in 
online surveys tends to be low -typically between 10% and 30%- and because it was simpler to 
interview a larger number of subjects than to ask each student to record a higher number of incidents. 
In this way, we avoid the risk of presenting a complicated and time-consuming survey, which would 
have an even more negative effect on participation. The main aim of this stage was to obtain a 
minimum of 200 critical incidents, a number that is considered theoretically adequate. Eventually, a 
total of 41 (21 men and 20 women) took part, reporting 392 critical incidents, of which 12 were 
rejected because they had not been correctly formulated. The sample thus comprised 380 valid 
critical incidents. 

Dear student. 

We are currently conducting a research project aimed at analyzing the factors that determine students’ 
perception on the quality of e-learning services. 

The first stage of this study requires the collaboration of a group of randomly-selected students. We would like to 
kindly ask for your collaboration in this project. All you have to do is answer this message reporting 5 positive 
and 5 negative incidents related with services provided to you by UOC. 

Please, consider the following examples of incidents that might serve you to understand the kind of opinions we 
are looking for. Imagine that you visit your local health centre. Some examples of incidents related with the 
quality of the service you receive might be: 

Positive: (1) “the person at the information desk was helpful’’, (2) ’’the doctor carried out a thorough 
examination”, ... 

Negative: (1) “I had spent too much time in the waiting room”, (2) “the doctor used a lot of technical terms that I 
could not understand”. 

We would like you to report specific situations, examples or experiences -either recent or from the past- 
regarding your relationship with the UOC. The situations can include any service provided by the UOC -not only 
the basic teaching service. Please, try to be as clear and specific as possible, avoiding abstract comments such 
as “the service was good”. 

Notice that all the information provided in this study will be completely confidential. 

Thank you very much for your participation and your assistance in this matter. 

Yours sincerely. 

Figure 3: E-mail sent to students 

The incidents were analyzed as follows: First, they were grouped according to type. For each type a 
sentence was written to describe the incident; both positive and negative incidents were included. 
Once all the critical incidents were grouped together the above process was repeated, now using the 
similarity between the sentences describing the incidents as the criterion. We thus obtained a 
hierarchical relation between critical incidents, their aggregate descriptions and, finally, the sections of 
quality. 
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The key stage in the process was the creation of two work teams. To monitor this process, 
responsibility for the task was assigned to two different teams (each of them composed by two 
professors from different knowledge areas): the first established the sections of quality following the 
generalization procedure described above and the second, using the sections established, directly 
assigned each of the critical incidents to one of these sections. The accuracy of the distribution 
process depends on the degree of agreement between the teams, that is, of the percentage of 
incidents that both place in the same section. Total agreement is represented by a score of 1. We 
obtained a score of 0.91, which should be considered satisfactory, taking into account that the 
literature considers an index of above 0.8 to be acceptable. Finally we checked the accuracy of the 
sections in order to determine whether they were able to define the construct of service quality in its 
entirety. To do this, we randomly extracted around 10% of the critical incidents (35) and then 
regrouped and reclassified the remaining incidents in the sections. We then re-assigned to these 
sections the 35 critical incidents that we had removed. 

Since we were able to place all these incidents in the sections, we concluded that they presented a 
reliable reflection of the constructed model. The categorization process highlighted the need to reject 
incidents which did not contain specific examples or experiences of the service received but reflected 
more general impressions, such as the advantages of online learning -time saving, availability, ease 
of access, the opportunity to combine studies and work, etc.- and its drawbacks -the feeling of 
isolation, the need to adapt to the environment, etc. (Juan et al. 2009). Among these comments, the 
opportunity to combine studies and family and professional life was highly valued by a part of the 
population who otherwise would not be able to study. After this process of refinement we had 350 
critical incidents, of which 184 were positive and 166 negative. Once the classification process was 
completed, and after checking the validity of the process, these critical incidents were finally grouped 
in 6 sections covering 33 definitions. Table 1 summarizes the sections or dimensions obtained. 


Table 1: Critical incidents reported by dimension 


Section or dimension 

Positive 

critical 

incidents 

Negative 

critical 

incidents 

Total number 
of critical 
incidents 

Percentage of 
critical 
incidents 

Learning process 

79 

60 

139 

40% 

Administrative processes 

31 

48 

79 

23% 

Teaching materials and resources 

26 

21 

47 

13% 

User’s interface 

23 

23 

46 

13% 

Relationships with the community network 

25 

5 

30 

9% 

Fees and compensations 

0 

9 

9 

3% 

Total 

184 

166 

350 

100% 


5. Analysis of results 

As can be seen in Table 1 the first dimension or section, learning process, accounts for about 40% of 
total critical impacts. This section represents the core service of any higher-education institution, and 
it includes categories such as the following ones: course design, learning planning, homework 
workload, instructors’ guidance and support, homework contribution to learning, instructors’ feedback, 
assessment system, instructors’ responsiveness, accuracy of responses and instructors’ courtesy. A 
total of 27 critical incidents were reported in this section referring to the guidance and support of the 
learning process by instructors. Some examples of these incidents are the following ones: “excellent 
guidance from instructors”, “some of the instructors in this course just sent an e-mail on each 
important deadline, but they did not perform any other action to encourage or guide students in the 
meantime”, etc. In this sense, students reported a large number of critical incidents (up to 9% being 
negative incidents) related with either the absence of personalized feedback from instructors. Another 
category that received a considerable number of critical incidents was the assessment process. From 
its 21 associated incidents, it can be derived that, on the average, UOC students have a positive view 
of our continuous assessment system, which “motivates to study regularly and to learn more about 
some advanced topics”. However, students also seem to have a negative view of the “excessive 
number of tests and homework activities that they need to complete in some courses”. Another two 
categories that can be considered as relevant in this section are those related to the course design 
and to the length of the response time-interval (i.e., the time between a student’s request submission 
and its corresponding instructor’s response). 
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The second section, in terms of relative importance, is the one related to administrative processes. It 
represents about 23% of the critical incidents reported, and it relates to the so-called ‘facilitating 
services’, that is, services that are indispensable for the rendering of the essential service even when 
they are not part of this essential service. In the case of our university, these services include all 
administrative processes, such as secretary’s office (registration, certification, prior learning 
assessment, etc.), the organization of face-to-face final tests, the handling and shipping of 
documentation (both of academic and administrative nature) and the professional performance of our 
administrative staff in terms of responsiveness, courtesy and accuracy of responses. 

The third section, teaching materials and resources, recorded 47 critical incidents (13% of the total 
number of incidents), 26 positive and 21 negative. According to the UOC pedagogical model, this 
section can be considered as being an integral part of the essential service. Nevertheless, it has 
been considered apart from the learning process because it has enough entity on its own. The 
section can be divided into the following three categories: contents (e.g. “it is good to use updated 
learning materials”, or “materials seem too superficial to me”), library (covering both resources 
availability and library general performance) and format. 

The fourth section, user’s interface, covers some service aspects related to the usability and technical 
performance of the UOC Virtual Campus and also of the university staff that provides online technical 
support to students and instructors. These categories are directly related with the fact that the kind of 
service being offered is an Internet-based one. Therefore, according to (Gronroos, 1990), it also 
reflects a facility service. The highest percentage of (positive) critical incidents in this section is found 
to be associated with the usability or browsing capabilities of the UOC Virtual Campus, that is, how 
simple and intuitive is the browsing inside the online learning environment (e.g. “Browsing the virtual 
learning environment is really easy”). The remaining two more representative categories of this 
section are those related to the reliability and connectivity levels of the UOC Virtual Campus. Some 
examples of positive critical incidents related to these categories are: “(...) everything works well: 
sending files and downloading materials (...)”, or “access to the campus was problem-free”. Also, 
some examples of negative critical incidents for these categories are: “I’ve suffered problems every 
time I’ve tried to download the materials for this course, especially at the beginning of the semester 
and Sunday evenings”, or “I’ve had difficulties when trying to connect and access the campus during 
the last days”. Within this section, only a small percentage of incidents (about 2%) refers to the 
responsiveness, accuracy of responses and courtesy of the technical staff. 

Students have also reported 30 critical incidents, most of them positive, related to the section 
relationship with the community network. This section constitutes a ‘supporting service’, that is, an 
optional service that contributes to differentiate our offer from the service being offered at other 
universities. In particular, UOC students give positive credit to “the possibility of interact with students 
from all over Spain and Latin-America, who are professionals working on jobs similar to mine”, and 
also to “the chance of making friends and working together in the learning process, performing online 
collaborative learning”. Finally, we have to consider the fees and compensation section. This section 
is considered by ZeithamI et al. (2004) as being particularly relevant when online consumers do not 
receive the expected service. In this section, only negative incidents were recorded (about 3% of the 
total number of incidents). In fact, some students perceived the courses as “expensive” and stated 
that they did not receive any refunds or compensations for missing or underperformed services. 

6. Lessons learned and recommendations to UOC managers 

Figure 4 summarizes the data in Table 1 and presents it in a more visual format. From both sources, 
and according to the previous discussion of results, we can establish the following recommendations 
for UOC managers. Notice that most of these suggestions could probably be extended to other 
universities worldwide offering online courses and degrees: 

■ First of all, total quality management in a university implies to have a holistic view of the service 
being offered, which includes not only the learning process section but also other dimensions. 
Nevertheless, the learning process is considered by students as the main quality dimension, since 
it constitutes the raison d’etre of the service being offered. 

■ Regarding the learning process, students give special attention to aspects such as: (a) the 
guidance and support received by instructors, (b) the assessment process, (c) helpfulness of the 
responses given by instructors to their requests, (c) associated response times, and (e) balanced 
workload and practical utility of the proposed homework activities. Therefore, these topics must 
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be carefully considered by the university and, in the case of our study, data seems to indicate that 
UOC students are generally satisfied with them. However, it becomes necessary to improve the 
feedback that our students receive from their instructors: students request a more personalized 
and complete feedback for each learning activity and not just a general feedback -as it is 
currently the case in most courses. Regarding the essential service, it is also important to 
highlight the section related to academic materials and resources. In this sense, the study results 
reinforce the idea that both materials and academic resources must be well-written, updated and 
especially suitable for being used in distance-education. Also, the library performance must be 
efficient and it must offer resources oriented to facilitate the online learning process. 

Positive and Negative incidents by Dimension 



Learning process Administrative Teaching materials User’s interface Relationships with Fees and 

processes and resources the communi^ compensations 

network 


Dimension 


o Positive Incidents D Negative Inciderts | 


Figure 4: A graphical representation of the number of incidents by section or dimension 

■ The section regarding administrative processes is the only one that shows more negative than 
positive critical incidents. Therefore, it is a section that requires priority attention. It is convenient 
not to forget that this section refers to services that are fully necessary for the correct 
development of the core service. Some categories need improvement in this section: responses 
from administrative staff should be more accurate, some administrative processes should be 
simplified and more transparent (that is, less bureaucratic), and handling and shipping of 
administrative documentations should be more efficient. A positive aspect of this section is the 
courtesy employed by administrative staff in their responses to students’ requirements. 

■ Regarding the user’s interface section, we can see that there are as much positive as negative 
incidents. Among the positive incidents, we can highlight the facility to browse the UOC Virtual 
Campus. Among the negative incidents, we can cite the reliability and connectivity problems that 
some students have suffered when accessing this Virtual Campus, especially at the beginning of 
each semester. 

■ Another positive aspect, from the students’ point of view, is the existence of an online learning 
community that provides support to the e-learning process and enriches it. This, in turn, 
contributes to reduce the risk of abandonment or drop-out and, at the same time, favours the 
creation of a real university community. 

■ Finally, university’s managers should work on the fees and compensation dimension, so that 
students do not perceive the service being received as an expensive one and, moreover, they can 
be compensated whenever any of the services they have paid for does not perform as well as 
expected. 

7. Conclusions 


This paper shows how students perceive online higher-education services and which quality sections 
or dimensions they consider important in their evaluation. The study applies a holistic conception of 
service quality, considering not only the core service (the learning process) but also the auxiliary 
administrative and backup services as well. Furthermore, based on the specific critical incidents 
reported by students, we have established some recommendations for university managers. On the 
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one hand, they should maintain and strengthen those aspects which have been referred to as positive 
critical incidents by students and, on the other hand, they should improve the quality of service from 
those reported as negative critical incidents. These actions will help to significantly improve the 
overall quality of service perceived by students and, consequently, students’ satisfaction with the 
services offered by the university. The results obtained are not only aimed at clarifying the 
determinants of perceived service quality in online higher education, but also show the advantages of 
the Critical Incident Technique over other exploratory inductive methods, particularly when research is 
conducted in online environments, as is the case analyzed in this article. This technique has been 
widely used to assess the underlying sources of satisfaction and dissatisfaction of consumers of 
services (Bitner et a!., 1990; Edvardsoon, 1988; 2001), but as far as we know it has never been 
applied before in the context of e-services (Sweeney and Lapp, 2004). We think that the use of this 
technique in online environments offers clear advantages over other qualitative techniques. For 
instance, organization of discussion groups is not easy to manage since students enrolled on online 
programs rarely meet face-to-face. Additionally, online students usually combine their courses with 
their professional activity, which do not give them too much free time to participate in discussion 
groups or face-to-face interviews. 
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